Reconfigure Linseed and linseed user when migrating from multi-index to single-index by asincu · Pull Request #5132 · tigera/operator

asincu · 2026-07-31T22:24:55Z

Description

Linseed will reconfigure its environment variables to set the correct backend for the
indices it is using and also set the base index name used for Cloud.
ES Kubecontrollers will stop running elasticsearch configuration
LogStorage Controllers Users will configure Linseed user instead of es-kubecontrollers.

Release Note

TBD

For PR author

Tests for change.
If changing pkg/apis/, run make gen-files
If changing versions, run make gen-versions

For PR reviewers

A note for code reviewers - all pull requests must have the following:

Milestone set according to targeted release.
Appropriate labels:
- kind/bug if this is a bugfix.
- kind/enhancement if this is a a new feature.
- enterprise if this PR applies to Calico Enterprise only.

Copilot

Pull request overview

This PR updates the operator’s log-storage reconciliation path for Calico Cloud index migration, shifting Elasticsearch user provisioning responsibilities and reconfiguring Linseed to use the correct single-index backend and index base names during migration.

Changes:

Add an IndexMigration / UseSingleIndex signal from bootstrap config through controller options into rendering logic.
Disable es-kube-controllers’ elasticsearchconfiguration controller during index migration and move user provisioning to the operator’s log-storage users controller.
Add Cloud single-index base index naming defaults and ensure deterministic ordering to avoid reconcile churn.

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
pkg/render/logstorage/linseed/linseed.go	Adds single-index backend env var rendering for single-tenant migration path.
pkg/render/kubecontrollers/kube-controllers.go	Adds `IndexMigration` flag and gates `elasticsearchconfiguration` controller enablement.
pkg/render/kubecontrollers/kube-controllers_test.go	Adds test asserting `elasticsearchconfiguration` is disabled during migration.
pkg/render/common/cloudconfig/cloudconfig.go	Extends `ToTenant` to optionally populate standard single-index base names deterministically.
pkg/render/common/cloudconfig/cloudconfig_test.go	Adds tests for `ToTenant(useSingleIndex)` behavior and stable ordering.
pkg/controller/utils/elasticsearch.go	Adds single-tenant username formatting and single-tenant Linseed/Dashboards user helpers.
pkg/controller/policyrecommendation/policyrecommendation_controller.go	Updates CloudConfig `ToTenant` call signature.
pkg/controller/options/options.go	Adds `IndexMigration` option plumbed from bootstrap config.
pkg/controller/manager/manager_controller_cloud.go	Updates CloudConfig `ToTenant` call signature.
pkg/controller/logstorage/users/users_controller.go	Enables operator user provisioning for single-tenant index migration; adds single-tenant user naming and credential “re-pointing”.
pkg/controller/logstorage/users/users_controller_test.go	Adds controller tests for single-tenant migrating behavior and credential repointing.
pkg/controller/logstorage/linseed/linseed_controller.go	Plumbs index-migration flag into Linseed render config and CloudConfig tenant conversion.
pkg/controller/logstorage/kubecontrollers/es_kube_controllers.go	Plumbs `IndexMigration` through controller state.
pkg/controller/logstorage/kubecontrollers/cloud.go	Sets kube-controllers render config `IndexMigration` during Cloud reconcile.
pkg/controller/logstorage/initializer/conditions_controller.go	Avoids no-op status writes, handles conflicts, and stabilizes condition ordering; includes users controller condition in migration.
pkg/controller/logstorage/dashboards/dashboards_controller.go	Updates CloudConfig `ToTenant` call signature and clarifies credential provisioning during migration.
pkg/controller/intrusiondetection/intrusiondetection_controller.go	Updates CloudConfig `ToTenant` call signature.
pkg/controller/compliance/compliance_controller.go	Updates CloudConfig `ToTenant` call signature.
pkg/common/discovery/discovery.go	Adds `INDEX_MIGRATION` bootstrap config parsing helper.
cmd/main.go	Reads `INDEX_MIGRATION` bootstrap config and sets controller options.
api/v1/tenant_types.go	Adds Cloud standard base index name mapping by datatype.

Copilot

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 1 comment.

Suppressed comments (3)

pkg/render/common/cloudconfig/cloudconfig.go:40

cloudStandardIndices uses a ComplianceBenchmarks base index name ("calico_compliance_benchmarks_results_standard") that doesn’t match the standard single-index name used elsewhere (e.g. Linseed rendering tests expect "calico_compliance_benchmarks_standard"). This will cause the generated Tenant indices/env vars to point Linseed (and user RBAC derived from these indices) at the wrong index base name.

	v1.DataTypeComplianceBenchmarks: "calico_compliance_benchmarks_results_standard",

pkg/render/common/cloudconfig/cloudconfig.go:48

cloudStandardIndices threat-feeds base index names (DomainSet/IPSet) don’t match the standard names used elsewhere (e.g. Linseed rendering tests expect "calico_threat_feeds_domain_set_standard" and "calico_threat_feeds_ip_set_standard"). If these names are wrong, Linseed/user RBAC will be configured for indices that don’t exist.

	v1.DataTypeThreatFeedsDomainSet: "calico_threatfeeds_domainnameset_standard",
	v1.DataTypeThreatFeedsIPSet:     "calico_threatfeeds_ipset_standard",

pkg/controller/logstorage/users/users_controller.go:370

clusterID() hard-codes the operator namespace as "tigera-operator". Use common.OperatorNamespace() (already used elsewhere in this controller) so this continues to work if the operator namespace is overridden.

	clusterIDConfigMapKey := client.ObjectKey{Name: "cluster-info", Namespace: "tigera-operator"}

cluster This will be set operator bootstrap config map and enabled when migrating from multi-index format to single-index format. Linseed will reconfigure its environment variables to set the correct backend for the indices it is using and also set the base index name used for Cloud.

…hile migrating to single-index storage Single-tenant clusters migrating to single-index storage need Linseed to hold RBAC for the new calico_* indices. es-kube-controllers cannot grant that, so during the migration the operator's log-storage users controller takes over user provisioning and es-kube-controllers stops running its elasticsearch configuration controller. - Run the users controller in single-tenant mode when IndexMigration is set, building the tenant configuration from the cloud config ConfigMap since single-tenant clusters have no Tenant resource. - Name the single-tenant Linseed and Dashboards users the way es-kube-controllers named them (<name>-<tenantID>-secure), and repoint existing credential secrets at those users while keeping their passwords, so credentials provisioned before the migration keep resolving. - Declare the standard single-index names on the Tenant that CloudConfig.ToTenant builds, gated on the caller opting in, so that clusters which are not migrating keep falling back to their existing index names. Sort the declared indices, as they are generated from a map. - Report the users TigeraStatus in the log-storage conditions aggregate while migrating.

…rom writing on every reconcile updateConditions built its result by ranging over the desiredConditions map, so the order of LogStorage.Status.Conditions was randomized on every reconcile. Conditions is an atomic list, so a reorder is a real change to the stored object: each reconcile bumped the resourceVersion, and since this controller also watches LogStorage, that re-enqueued itself. The write loop ran continuously, and reconciles fired faster than the informer cache could converge - so reconciles read a stale tigera-secure and their status updates were rejected with "the object has been modified". Sort the conditions by type so the stored list is stable, skip the status update entirely when the computed conditions match what is already stored, and requeue instead of erroring when an update does hit a conflict.

…nt clusters Calico Cloud single-tenant clusters had their Elasticsearch users provisioned by es-kube-controllers, and the operator only took over while migrating to single-index storage. Take over for all of them, and grant Linseed access to the indices its cluster actually stores data in rather than to both name formats. - Run the log-storage users controller for every Calico Cloud single-tenant cluster, and stop es-kube-controllers running its elasticsearch configuration controller there so that it does not overwrite the users we own. Wait for the cloud config ConfigMap rather than erroring when it is not there yet. - Derive the Linseed role's index privileges from the tenant: the declared base index names when the cluster stores data in single-index format, the multi-index names otherwise - dropping the tenant qualifier for clusters on their own Elasticsearch, whose indices do not carry it. - Report the users TigeraStatus in the log-storage conditions aggregate for Calico Cloud rather than only while migrating. - Rename the index migration knob to USE_SINGLE_INDEX / UseSingleIndex, matching the naming already used by the linseed controller and render code. - Move CloudStandardIndices out of the API module and unexport it, as it is only consumed when building the single-tenant Tenant from the cloud config.

- Skip indices with an empty base index name when building the Linseed role's index privileges. A Tenant declaring an index without a base name - whether misconfigured, or carrying a DataType added later without a mapping - would otherwise be wildcarded into "*", granting Linseed access to every index in Elasticsearch. Fall back to Linseed's default calico_ names when no usable base index name remains, which is what a Tenant declaring no indices at all already got. - Fix a comment on the single-tenant Linseed backend, which described the branch as migrating to multi-tenant style indices when it is gated by UseSingleIndex and migrating to single-index storage. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

asincu requested a review from a team as a code owner July 31, 2026 22:24

marvin-tigera added this to the v1.44.0 milestone Jul 31, 2026

marvin-tigera added docs-pr-required release-note-required labels Jul 31, 2026

asincu added the docs-not-required label Jul 31, 2026

marvin-tigera removed the docs-pr-required label Jul 31, 2026

asincu removed release-note-required docs-not-required labels Jul 31, 2026

marvin-tigera added docs-pr-required release-note-required labels Jul 31, 2026

asincu added release-note-not-required and removed docs-pr-required release-note-required labels Jul 31, 2026

marvin-tigera added the docs-pr-required label Jul 31, 2026

asincu added docs-not-required and removed docs-pr-required labels Jul 31, 2026

asincu requested a review from Copilot August 4, 2026 19:50

Copilot started reviewing on behalf of asincu August 4, 2026 19:50 View session

Copilot AI reviewed Aug 4, 2026

View reviewed changes

Comment thread pkg/render/kubecontrollers/kube-controllers.go

Comment thread api/v1/tenant_types.go Outdated

Comment thread pkg/controller/options/options.go Outdated

Comment thread pkg/render/logstorage/linseed/linseed.go Outdated

asincu requested a review from Copilot August 5, 2026 20:11

Copilot started reviewing on behalf of asincu August 5, 2026 20:11 View session

Copilot AI reviewed Aug 5, 2026

View reviewed changes

Comment thread pkg/controller/utils/elasticsearch.go

asincu added 4 commits August 5, 2026 13:18

(fix): Use CalicoEnterprise instead of TigeraEnterprise

96dc123

asincu force-pushed the index_migration_backport branch from 71d121b to 6913b70 Compare August 5, 2026 20:20

asincu force-pushed the index_migration_backport branch from 6913b70 to cba9c23 Compare August 5, 2026 20:48

Navigation Menu

Uh oh!

Reconfigure Linseed and linseed user when migrating from multi-index to single-index - #5132

Reconfigure Linseed and linseed user when migrating from multi-index to single-index#5132
asincu wants to merge 6 commits into
masterfrom
index_migration_backport

asincu commented Jul 31, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

asincu commented Jul 31, 2026

Description

Release Note

For PR author

For PR reviewers

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants